RGB-D Mapping: Using Depth Cameras for Dense 3D Modeling of Indoor Environments

RGB-D cameras are novel sensing systems that capture RGB images along with per-pixel depth information. RGB-D cameras rely on either structured light patterns combined with stereo sensing [6,10] or time-of-flight laser sensing [1] to generate depth estimates that can be associated with RGB pixels. Very soon, small, high-quality RGB-D cameras developed for computer gaming and home entertainment applications will become available at cost below $100. In this paper we investigate how such cameras can be used in the context of robotics, specifically for building dense 3D maps of indoor environments. Such maps have applications in robot navigation, manipulation, semantic mapping, and telepresence. The robotics and computer vision communities have developed a variety of techniques for 3D mapping based on laser range scans [8, 11], stereo cameras [7], monocular cameras [3], and unsorted collections of photos [4]. While RGB-D cameras provide the opportunity to build 3D maps of unprecedented richness, they have drawbacks that make their application to 3D mapping difficult: They provide depth only up to a limited distance (typically less than 5m), depth values are much noisier than those provided by laser scanners, and their field of view (∼ 60◦) is far more constrained than that of specialized cameras or laser scanners typically used for 3D mapping (∼ 180◦). In our work, we use a camera developed by PrimeSense [10]. The key insights of this investigation are: first, that existing frame matching techniques are not sufficient to provide robust visual odometry with these cameras; second, that a tight integration of depth and color information can yield robust frame matching and loop closure detection; third, that building on best practice techniques in SLAM and computer graphics makes it possible to build and visualize accurate and extremely rich 3D maps with such cameras; and, fourth, that it will be feasible to build complete robot navigation and interaction systems solely based on cheap depth cameras.